Personal video recorder

ABSTRACT

A sequence of picture slices is encoded as reference slices and non-reference slices, wherein the reference slices include B slices, by forming, for each B slice, at least one data packet containing data values derived from brightness and color information pertaining to the slice. The data packet for each B slice includes a header element indicating whether the B slice is a reference slice. The header element of each reference B slice has a value that depends on depth of the reference B slice in a hierarchy of discardability.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of co-pending U.S. patent applicationSer. No. 15/926,931 filed on Feb. 12, 2020, which is a continuation ofU.S. patent application Ser. No. 15/370,981 filed on Dec. 6, 2016, nowU.S. Pat. No. 9,961,359, which is a continuation of U.S. patentapplication Ser. No. 13/955,372 filed on Jul. 31, 2013, now U.S. Pat.No. 9,549,179, which is a continuation of U.S. patent application Ser.No. 12/183,055, now U.S. Pat. No. 8,526,489 issued on Sep. 3, 2013,which claims the benefit of U.S. Provisional Application No. 60/972,295filed Sep. 14, 2007, the contents of which are each incorporated hereinby reference for all purposes.

BACKGROUND OF THE INVENTION

The subject matter disclosed in this application relates to a personalvideo recorder and, in particular, to a method and apparatus forimproving trick playback operation of a personal video recorder.

A television programming provider typically produces a continuous set ofprogramming signals (also known as “network feeds”) for distribution bya service provider over a video transmission network to a wide audienceof viewers. Conventionally, the programming signal begins as anuncompressed video sequence and at least one corresponding uncompressedaudio sequence. The uncompressed video sequence consists of a series ofsequential pictures and is assembled at a production facility. Afterassembly, the uncompressed video sequence is compressed by a videoencoder, which encodes each picture and creates a corresponding codedpicture (also known as an access unit). Any corresponding audiosequences are compressed by an audio encoder. The coded audio and videosequences are transmitted over the transmission network to customerpremises at which the audio and video sequences for a selected programare decoded and presented to the viewer.

ISO/IEC 14496-10 (MPEG-4 part 10) Advanced Video Coding (AVC), commonlyreferred to as H.264/AVC, prescribes a standard for coding image datafor transmission and storage. H.264/AVC defines a frame as containing anarray of luma samples and two corresponding arrays of chroma samples andas being composed of two fields, a top field and a bottom field. A 16×16block of luma samples and two corresponding blocks of chroma samples isreferred to as a macroblock. A picture (a generic term for a field or aframe) is partitioned into slice groups and each slice group containsone or more slices, each of which in turn contains an integer number ofmacroblocks.

H.264/AVC defines an I slice, a P slice and a B slice. Each slice isencoded as blocks of transform coefficients. The definition of an Islice in H.264/AVC is generally accepted as meaning a slice that isdecoded using prediction only from decoded samples within the sameslice, i.e. an I slice is self-contained. Similarly, under H.264/AVC, aP slice is a slice that may be decoded using prediction from decodedsamples within the same slice or from decoded samples of at most onepreviously decoded reference picture using at most one motion vector andreference index to predict the sample values of each block. Thus, eachblock of transform coefficients in a P slice relies on only onepreviously decoded reference picture. And under the generally acceptedinterpretation of H.264/AVC, a B slice is a slice that may be decodedusing prediction from decoded samples within the same slice or fromdecoded samples of at least one reference picture using at most twomotion vectors and reference indices to predict the samples of eachblock. Thus, each block of transform coefficients in a B slice may relyon two reference pictures. Although any block of a P slice relies ononly one reference picture, different blocks in a given P slice may relyon different reference pictures. Similarly, although any block of a Bslice may rely on only two reference pictures, different blocks in agiven B slice may rely on different reference pictures. Each slice has aslice header containing a slice_type syntax element, indicating whetherthe slice is an I slice, a P slice or a B slice, and a reference picturelist indicating the pictures, if any, on which the slice relies fordecoding.

A picture that contains only I slices may be referred to as an Ipicture. Similarly, a picture that contains only I slices and P slicesmay be referred to as a P picture and a picture that contains one ormore B slices may be referred to as a B picture. H.264/AVC allows I, Pand B pictures to be used as reference pictures.

The image information in each picture is represented by data containedin one or more Network Abstraction Layer (NAL) units. There are twotypes of NAL units, namely Video Coding Layer (VCL) NAL units andnon-VCL NAL units. The subject matter of this application relates to theVCL NAL units and accordingly subsequent references to NAL units shouldbe interpreted as referring to VCL NAL units. A NAL unit is a packethaving an integer number of bytes and contains the image information forone slice. The first byte of a NAL unit is a header that contains atwo-bit syntax element nal_ref_idc. H.264/AVC specifies that nal_ref_idcis zero for a slice that is part of a non-reference picture and is notequal to zero for a slice of a reference picture, and that whennal_ref_idc is equal to zero for one slice of a particular picture, itshall be equal to zero for all slices of that picture. Thus, for anygiven picture, the nal_ref_idc values for all the slices are zero or allare non-zero. Accordingly, it is meaningful to refer to a picture forwhich nal_ref_idc=0 and to a picture for which nal_ref_idc≠0. AlthoughH.264/AVC does not use the terms “reference slice” and “non-referenceslice,” it is convenient to use these terms to refer, respectively, to aslice for which nal_ref_idc≠0 and a B slice for which nal_ref_idc=0.

An AVC encoder receives an input frame for encoding and generates abitstream representing, for each slice, the slice header and a set oftransform coefficients. The mode of operation of a suitable AVC encoderis well understood by those skilled in the art. The bitstream generatedby the AVC encoder is passed to a network abstraction layer, which formsthe NAL units with the required syntax elements (including thenal_ref_idc bits) at the proper location (NAL header) in the NAL units.

Signals encoded using H.264/AVC are widely used for distributingtelevision program material over various types of networks, includingcable, IP TV and satellite using various protocols for encapsulating theNAL units. For example, Internet protocol is used for IP TV whereas theMPEG-2 transport stream (as defined in ISO 13818-1) is used in cable andsatellite networks as a robust means for delivering a signal encoded inaccordance with H.264/AVC. An MTS that delivers just one program (videoand associated audio) is referred to as a single program transportstream (SPTS) whereas an MTS that delivers more than one program isreferred to as a multi-program transport stream (MPTS).

In the case of an MTS based distribution system, the network abstractionlayer places the NAL units in a video packetized elementary stream(video PES) and supplies the video PES to an MPEG-2 transport stream(MTS) layer. The MTS layer includes a multiplexer that selects the videoPES and an associated audio PES, and video and audio PESs of otherprograms, in the sequence that is required in order to form MPTSpackets.

The data bits of the MPTS packets are used to encode a signal fortransmission over a channel to a receiver at which the data bits arerecovered from the received signal and passed to an MTS layer whichparses the bitstream and selects the video PES and audio PES of adesired program and supplies the video PES packets to an AVC decoder andthe audio PES packets to an audio decoder. The AVC decoder includes anetwork abstraction layer that extracts the NAL units from the video PESpackets. The AVC decoder calculates a set of transform coefficients fromthe NAL unit bitstream and processes the transform coefficients and anymotion vectors in inverse fashion to the operations in the AVC encoderto create a decoded frame corresponding to the input frame that waspresented for encoding. The decoded frame is loaded into a video displaybuffer. Decoded frames are read from the display buffer at the properconstant rate and are presented for display at the output of the AVCdecoder.

The AVC decoder includes a decoder buffer for temporarily storingreference slices so that they will be available for decoding laterdependent slices. The nal_ref_idc value allows the AVC decoder todetermine readily whether a particular slice should be stored(nal_ref_idc≠0) or may be discarded (nal_ref_idc=0).

It is conventional to organize a sequence of pictures as a GOP, or groupof pictures, having a repeating structure of I, P and B pictures. Inimplementations of the MPEG-2 standard, the GOP may comprise 12 picturesin the sequence IBBPBBPBBPBB (or 15 pictures in the sequenceIBBPBBPBBPBBPBB) whereas implementations of H.264/AVC may employ ahierarchical GOP structure in the form IBBBPBBBP etc. orIBBBBBBBPBBBBBBBP etc., depending on whether the decoder stores one ortwo B pictures. The picture at the beginning of the GOP is sometimes aninstantaneous decoding refresh (IDR) picture, or an I or P picture.Accordingly, a GOP is usually self contained: a picture in an earlierGOP usually does not serve as reference for a picture in a later GOP.H.264/AVC does not differentiate among reference slices based on thenon-zero value of the nal_ref_idc syntax element. IP based systemssometimes use the three available non-zero values of the nal_ref_idcsyntax element to signal a priority level for the NAL units so that IPpackets containing NAL units with nal_ref_idc=3 are handled with ahigher priority than those containing NAL units with nal_ref_idc valuesequal to 2 or 1. It has also been proposed that a scalable video coding(SVC) extension of H.264/AVC should employ the non-zero values of thenal_ref_idc syntax element to distinguish among temporal levels ofpictures. MTS based applications do not currently use the non-zerovalues of the nal_ref_idc syntax element to differentiate the handlingof NAL units for reference slices.

Many subscribers to cable and satellite television distribution servicesuse PVRs (personal video recorders) to record television programmaterial for later playback and viewing. In this case, the signals arestored in coded form and are played back when desired and decoded insimilar manner to the stand alone decoder described above.

In normal operation of the PVR, the video and audio PES packets for aselected program are temporarily saved in a suitable memory device, suchas a hard disk drive. When a saved program is selected for viewing bythe user, the audio and video PES packets are read from the memorydevice. The video PES packets are supplied to the AVC decoder and theaudio PES packets are supplied to an audio decoder, as described above.The AVC decoder supplies the decoded frames to the video display bufferand the frames are read from the display buffer for presentation to theviewer.

A typical PVR supports various trick playback modes, including fastforward (FF) and rapid reverse (RR), which allow a viewer to scanrapidly through material of little interest. The PVR accomplishes FF andRR playback by discarding pictures of the received sequence, i.e. byomitting pictures of the received sequence from the sequence that isdecoded and supplied to the video display buffer. The PVR displayspictures at the normal constant rate (i.e. about 30 frames per second inthe United States) but since pictures of the received sequence arediscarded, the displayed image evolves at a greater speed than that innormal playback. For example, if the PVR discarded every other pictureduring FF playback, the displayed image would evolve at twice normalplayback speed.

In FF playback, the pictures that are retained in the sequence arepresented in the same order as in normal playback. In the case ofreverse playback, further manipulation is necessary so that picturesreceived later in the sequence will be available for presentation beforepictures that were received earlier.

In order to minimize degradation of the displayed image during FF or RRplayback, it is desirable that the discarded pictures not be referencepictures, since reference pictures are required to decode the dependentslices. This requirement can be applied readily to a signal encodedusing MPEG-2 by discarding B pictures, because under MPEG-2, a B pictureis not used as a reference picture. If all B pictures in an MPEG-2sequence employing the standard GOP structure IBBPBB etc. werediscarded, the FF or RR speed would be three times normal playbackspeed. In principle, this approach could be applied to a signal encodedusing H.264/AVC, by discarding non-reference B pictures. In this case,no reference slices would be discarded and all dependent slices could beproperly decoded. However, in a practical implementation of H.264/AVCthe macroblocks in a B slice may refer to as many as five picturesincluding reference B pictures and therefore there may be relatively fewpictures in a given H.264/AVC sequence for which nal_ref_idc=0.Accordingly, in order to achieve FF and RR playback, particularly atspeeds from three to six times normal playback speed, it may benecessary to discard reference pictures and the displayed image mayaccordingly be degraded to an undesirable extent.

SUMMARY OF THE INVENTION

According to a first aspect of the present invention there is provided amethod of creating a bitstream in which a sequence of picture slices isencoded as reference slices and non-reference slices, wherein thereference slices include B slices, said method comprising, for each Bslice, forming at least one data packet containing data values derivedfrom brightness and color information pertaining to the slice, and foreach B slice, including in at least said one data packet a headerelement indicating whether the B slice is a reference slice, and whereinthe header element of each reference B slice has a value that depends ondepth of the reference B slice in a hierarchy of discardability.

According to a second aspect of the present invention there is providedapparatus for creating a bitstream in which a sequence of picture slicesis encoded as reference slices and non-reference slices, wherein thereference slices include B slices, said apparatus comprising a means forperforming a method comprising, for B each slice, forming at least onedata packet containing data values derived from brightness and colorinformation pertaining to the slice, and for each B slice, including inat least said one packet a header element reflecting whether the B sliceis a reference slice, and wherein the header element of each reference Bslice has a value that depends on depth of the reference B slice in ahierarchy of discardability of reference B slices.

According to a third aspect of the present invention there is provided)a method of decoding a bitstream conveying a sequence of pictures eachcontaining at least one picture slice, wherein each picture slice isencoded as a reference slice or a non-reference slice, the referenceslices include B slices, the bits of each reference B slice are conveyedby at least one data packet that includes a header element having avalue that depends on depth of the reference B slice in a hierarchy ofdiscardability, and said method comprises receiving a trick play commandcalling for pictures to be discarded without being presented, andselecting a picture to be discarded based on the value of the headerelement.

According to a fourth aspect of the present invention there is providedapparatus for decoding a bitstream conveying a sequence of pictures eachcontaining at least one picture slice, wherein each picture slice isencoded as a reference slice or a non-reference slice, the referenceslices include B slices, and the bits of each reference B slice areconveyed by at least one data packet that includes a header elementhaving a value that depends on depth of the reference B slice in ahierarchy of discardability, the apparatus comprising a means forperforming a method comprising receiving a trick play command callingfor pictures to be discarded without being presented, selecting areference B picture to be discarded based on the value of the headerelement.

According to a fifth aspect of the present invention there is provided acomputer readable medium containing instructions that, when executed bya computing machine receiving a bitstream in which a sequence of pictureslices is encoded as reference slices and non-reference slices, whereinthe reference slices include B slices, cause the computing machine toperform a method comprising, for each B slice, forming at least one datapacket containing data values derived from brightness and colorinformation pertaining to the slice, and for each B slice, including inat least said one data packet a header element indicating whether the Bslice is a reference slice, and wherein the header element of eachreference B slice has a value that depends on depth of the reference Bslice in a hierarchy of discardability.

According to a sixth aspect of the present invention there is provided amethod of assisting in trick play operation of a video recording andplayback device, comprising, at a transmitting location, encoding asequence of pictures, each containing at least one picture slice,selectively in either a normal mode or a trick play assist mode, and inthe trick play assist mode, encoding each slice as one of at least twodifferent types depending on prediction needed to decode the slice, foreach slice of at least one type, forming at least one data packetcontaining data values derived from image information pertaining to theslice, creating at least one data value for each slice of said one typethat indicates depth of the picture containing the slice in a hierarchyof discardability, encapsulating the data packets in a transport streamwith the created data values, and transmitting the transport stream to areceiving location, at the receiving location, extracting the datapackets from the transport stream and temporarily storing the datapackets for later playback in normal operation or trick play operation,and in trick play operation, detecting whether the sequence of pictureslices was encoded in the trick play assist mode and, if so, selectivelydiscarding pictures depending on said data values.

According to a seventh aspect of the present invention there is provideda method of creating a bitstream in which a sequence of picture slicesis encoded as slices of at least two different types depending onprediction needed to decode the slice, said method comprising, for eachslice of at least one type, forming at least one data packet containingdata values derived from brightness and color information pertaining tothe slice, and for a slice of said one type, including in at least saidone data packet a header element indicating whether the slice is areference slice, and wherein in the event that said slice is a referenceslice the header element has a value that depends on depth of thereference slice in a hierarchy of discard ability.

BRIEF DESCRIPTION OF THE DRAWINGS

For a better understanding of the invention, and to show how the samemay be carried into effect, reference will now be made, by way ofexample, to the accompanying drawings, in which:

FIG. 1 is a block schematic diagram of an audio-video encoder embodyingthe present invention,

FIG. 2 is flow chart illustrating a first personal video recorderembodying the present invention,

FIG. 3 is a block schematic diagram of a second personal video recorderembodying the present invention,

FIG. 4 is a flow chart illustrating a decision tree that may be executedby an AVG encoder included in the audio-video encoder illustrated inFIG. 1, and

FIG. 5 is a block schematic diagram of a computing machine that may beused to implement parts of the AVC encoder.

DETAILED DESCRIPTION

H.264/AVC does not restrict the type (I, P or B) of pictures to which aB picture may refer. Since a B picture may serve as reference foranother B picture, and the latter B picture may serve as reference for afurther B picture, we may describe a hierarchy of dependent pictures,i.e. pictures that may rely on one or more reference pictures fordecoding, in ascending order of importance:

1. I pictures.

2. P pictures.

3. Reference B pictures that use only I and/or P pictures as referencepictures.

4. Reference B pictures that use B pictures in level 3 of the hierarchy(and may also use I and/or P pictures) as reference pictures.

5. Reference B pictures that use B pictures in level 4 of the hierarchy(and may also use I and/or P pictures and B pictures in level 3) asreference pictures.

6. Non-reference B pictures.

In this hierarchy of importance, 1 is the highest level (most important)and 6 is the lowest level (least important). The higher a picture'slevel in the hierarchy of importance, the more important is it tosuccessful decoding. Thus, if a B picture at a lower level in thehierarchy were discarded, the playback image would be less seriouslydegraded than if a B picture at a higher level in the hierarchy werediscarded.

We may use this hierarchy of importance of dependent pictures toestablish rules of priority that may be used by a PVR for determiningpictures that should be discarded in trick modes to achieve differenttrick mode playback rates. For example:

First level of discardability: non-reference B pictures (level 6 inhierarchy of importance).

Second level of discardability: non-reference B pictures plus referenceB pictures in level 5.

Third level of discardability: non-reference B pictures plus reference Bpictures in levels 5 and 4.

Fourth level of discardability: non-reference B pictures plus referenceB pictures in levels 5, 4 and 3.

Fifth level of discardability: some P pictures plus all B pictures.

Sixth level of discardability: some I pictures plus some P pictures andall B pictures.

It will be appreciated that at each succeeding level of discardability,from the first (or highest) level to the sixth (or lowest) level, agreater number of pictures is discarded and accordingly a higherplayback rate is achieved.

If a viewer's command calls for a 3× playback rate, such that two thirdsof the pictures must be discarded, it will typically be necessary todiscard all the pictures in the first through fourth levels, whereas ifthe command calls for a 2× rate, it might only be necessary to discardthe pictures in the first and second levels.

FIG. 1 illustrates an audio-video encoder 10 that may be located at thepremises of a television programming service provider, such as a cableoperator. The encoder 10 includes an AVC encoder 12 having two distinctmodes of operation. Thus, the AVC encoder may operate in theconventional manner in which dependent pictures are encoded withoutregard to discardability of the pictures but it may alternativelyoperate having regard to discardability of pictures in order to assisttrick play operation of a PVR that receives the encoded pictures.

The AVC encoder 12 receives frames and encodes the frames usingH.264/AVC based on a GOP structure. It will be recalled from thedefinition of an AVC hierarchial GOP that in each set of pictures in aGOP, the first picture is an I or P picture and the subsequent picturesare B pictures, with the number of B pictures depending on thehierarchy. For each dependent picture within a GOP, the AVC encoder 12determines which pictures to use as reference for encoding the currentpicture based on the importance of dependent pictures that have alreadybeen encoded. The encoder limits the reference B pictures to threelevels of importance and requires that a picture in a given level ofimportance not be used to create a picture at a higher level ofimportance. For example, the encoder may use a B picture in level 3 tocreate a picture in level 4 but may not use a B picture in level 3 tocreate a picture in level 2 or level 1, and the encoder may not use a Bpicture to create a P picture.

The AVC encoder assigns a nal_ref_idc value to a reference B picturebased on the importance of the picture and enters the assignednal_ref_idc value in the nal_ref_idc syntax element of the (or each) NALunit of the picture. Preferably, nal_ref_idc=1 is assigned to referenceB pictures in level 4, nal_ref_idc=2 is assigned to reference B picturesin level 3, and nal_ref_idc=3 is assigned to reference B pictures inlevel 2. B pictures with nal_ref_idc=2 are not used as referencepictures for predicting B pictures for which nal_ref_idc=3, and Bpictures with nal_ref_idc=1 are not used for predicting B pictures forwhich nal_ref_idc=3 or 2. However, a B picture with nal_ref_idc=1 may beused for predicting a non-reference B picture. Then, the priority ofdiscardability rules can be expressed:

First level of discardability: B pictures for which nal_ref_idc=0.

Second level of discardability: B pictures for which nal_ref_idc=0 or 1.

Third level of discardability: B pictures for which nal_ref_idc=0, 1 or2.

Fourth level of discardability: all B pictures.

Fifth level of discardability: some P pictures plus all B pictures.

Sixth level of discardability: some I pictures plus some P pictures andall B pictures.

The encoded bits are passed to the network abstraction layer to form theNAL units, the NAL units are placed in the video PES, and the video PESpackets are passed to the MTS layer 14.

The AVC encoder also generates supplementary assist information thatindicates whether the AVC encoder is operating in the conventional modeor in the trick play assist mode and provides this assist information,which may be a single bit, to the MTS layer. The MTS layer places theassist information as a descriptor (for example as an extension to theadaptation_field_data_descriptor specified in DVB_SI) in the MPEG-2multiplex.

FIG. 2 illustrates a PVR that may be located at a customer premise. ThePVR receives the MPTS containing the coded pictures and associated codedaudio, and the assist information, and the MTS layer 20 supplies thevideo and audio PES packets (and assist information) for a selectedprogram to a memory device 22, such as a hard disk drive, which storesthe packets and assist information until the viewer selects the programfor viewing. The video and audio PES packets and the assist informationare then read from the memory device and the video PES packets andassist information are supplied to an AVC decoder 24 whereas the audioPES packets are supplied to an audio decoder 26. The AVC decoderextracts the NAL units from the video PES packets. At normal playbackspeed, the AVC decoder operates in inverse fashion to the AVC encoderand creates decoded frames that are loaded into a display buffer, asdescribed above, and are subsequently read from the display buffer atthe proper constant rate and are presented to the viewer.

In trick play mode, the AVC decoder receives a message indicating theplayback rate requested by the viewer. The AVC decoder checks whetherthe assist information indicates that the program has been encoded usingthe trick play assist mode and, if so, parses the bitstream in order torecover the NAL units and extract the respective nal_ref_idc values anduses the nal_ref_idc values of the B pictures to select the pictures todiscard. The AVC decoder discards sufficient pictures, starting with thehighest level of discardability, that when the remaining pictures arepresented at the same constant rate the requested playback rate isachieved. For example, the AVC decoder may create a table for thecurrent GOP containing the number of B pictures for each value ofnal_ref_idc, determine the highest level of discardability that willresult in the proper number of pictures being discarded, and discard thepictures that meet this level of discardability. In this manner, at agiven playback rate the pictures that are discarded are not needed todecode the pictures that are retained and accordingly degradation of thedisplayed image is minimized. Thus, the non-zero values of thenal_ref_idc syntax element allow the video decoder to make a moreintelligent decision regarding which pictures to omit from the sequenceloaded into the display buffer.

The subject matter disclosed in this application is not restricted tothe case in which the encoding in H.264/AVC takes place at the premisesof the television programming service provider. For example, theprogramming services provider may transmit video that has been encodedin MPEG-2 and the PVR may transcode the MPEG-2 video to H.264/AVC forstoring because of the superior compression offered by H.264/AVC. Thus,referring to FIG. 3, the MPEG-2 video data is decoded by an MPEG-2decoder 30 and then encoded to H.264/AVC. The H.264/AVC encoder 32operates in essentially the manner described with reference to FIG. 1,except that the MTS layer (not shown in FIG. 3) produces a singleprogram transport stream that is not transmitted over a channel but isstored in the memory device 22. Upon playback, the PVR operates in themanner described with reference to FIG. 2.

The encoder described with reference to FIG. 1 makes a decisionregarding the level of importance of a dependent picture based on thepictures that have already been encoded and are available as referencepictures. In another encoder, which is topologically the same as thatshown in FIG. 1, the encoder determines the importance of a B pictureafter encoding and assigns a value of nal_ref_idc accordingly. In thiscase, the AVC encoder 12 identifies the type of each slice in thepicture based on the slice_type syntax element of the slices andidentifies the pictures that the slice refers to based on the referencepicture list in the slice header. The encoder 12 creates a table thatcontains, for each B picture, an entry that identifies each picturereferred to by the slices in the picture. Referring to FIG. 4, for eachB slice in the B picture (selected by decision 40), the encoder assignsa value to the syntax element nal_ref_idc based on the entries in thetable. For example, the AVC encoder first determines (decision 42)whether the picture that contains the slice is a reference picture. Ifthe picture is not a reference picture, i.e. the picture is not referredto in the reference picture list for any slice of any other picture inthe GOP, the video encoder sets nal_ref_idc=0 for each slice in thepicture, consistently with H.264/AVC. If the picture that contains theslice is a reference picture, the encoder then determines (decision 44)whether the picture refers only to I or P pictures and, if so setsnal_ref_idc=3. In similar fashion, as shown in FIG. 4, the encoder setsnal_ref_idc=2 if the picture that contains the slice refers only to an Ior P picture and a picture for which nal_ref_idc=3 and otherwise setsnal_ref_idc=1 (decision 46).

In the case of the embodiments described in connection with FIGS. 1 and2 the video PES packets are conveyed to the PVR in an MPEG-2 transportstream. It will be understood by those skilled in the art that MPEG-2transport stream packets are of fixed length, each 188 bytes long andhaving a four byte header. Accordingly, when H.264/AVC NAL units areencapsulated in MTS packets, each NAL unit is spread over numerous MTSpackets and in order to identify the beginning of a new NAL unit and theend of a previous NAL unit in the MPTS packets, each NAL unit ispreceded by a start code prefix of three bytes; emulation of a startcode prefix within a NAL unit is prevented by including emulationprevention bytes in the NAL unit byte stream. However, the invention mayalso be applied to IP TV, in which the AVC NAL units are conveyed in IPpackets. IP packets are of variable length and a single IP packet mayencapsulate a complete NAL unit, and it is not necessary to include astart code prefix or emulation prevention bytes.

Referring to FIG. 5, suitable television programming service providerpremise equipment may be implemented using a computer 50 comprising oneor more processors 51, random access memory 52, read-only memory 53, I/Odevices 54, a user interface 55 and a hard disk drive 58, configured ina generally conventional architecture. The computer operates inaccordance with a program that is stored in a computer readable medium,such as the hard disk drive 58 or a CD-ROM 57, and is loaded into therandom access memory 52 for execution. The program is composed ofinstructions such that when the computer receives a bitstreamrepresenting a sequence of picture slices encoded as reference slicesand non-reference slices, by way of a suitable interface included in theI/O devices 54, the computer allocates memory to appropriate buffers andutilizes other suitable resources and functions to perform the variousoperations that are described above as being performed by the encoder.

It will be appreciated by those skilled in the art that the programmight not be loadable directly from the CD-ROM 57 into the random accessmemory utilizing the CD-ROM drive 56 and that generally the program willbe stored on the CD-ROM or other distribution medium in a form thatrequires the program to installed on the hard disk drive 58 from theCD-ROM 57.

The subject matter described with reference to FIGS. 1 and 2 is notlimited to the case in which the AVC encoder has two modes of operation.The AVC encoder may be operated at all times in the trick play assistmode, in which case there is no need to include the assist informationin the MTS multiplex because the AVC decoder always operates in themanner described with reference to FIG. 2.

In some implementations of H.264/AVC the sequence of pictures iscomposed only of I pictures and P pictures, i.e. there are no Bpictures. In this case, non-zero values of nal_ref_idc may be used tofacilitate decisions regarding discarding of P pictures in trick playmode of a PVR.

In addition, the subject matter disclosed in this application is notrestricted to use of the non-zero nal_ref_idc values to facilitatedecision making over pictures to discard in trick playback mode of aPVR. For example, the non-zero values may be encoded to signal otherinformation to facilitate operation of a PVR.

It will be appreciated that the invention is not restricted to theparticular embodiment that has been described, and that variations maybe made therein without departing from the scope of the invention asdefined in the appended claims, as interpreted in accordance withprinciples of prevailing law, including the doctrine of equivalents orany other principle that enlarges the enforceable scope of a claimbeyond its literal scope. Unless the context indicates otherwise, areference in a claim to the number of instances of an element, be it areference to one instance or more than one instance, requires at leastthe stated number of instances of the element but is not intended toexclude from the scope of the claim a structure or method having moreinstances of that element than stated. The word “comprise” or aderivative thereof, when used in a claim, is used in a nonexclusivesense that is not intended to exclude the presence of other elements orsteps in a claimed structure or method.

The invention claimed is:
 1. A method of decoding a plurality of codedpictures provided in a bitstream, the method comprising: identifyingeach of the plurality of coded pictures as being one of a referencepicture or nonreference picture; determining a hierarchy of datadependency where each of the coded pictures is located, wherein at leastone level in the hierarchy of data dependency identifies discardablepictures being coded pictures associated with a nal_ref_idc field equalto 0, and at least one level in the hierarchy of data dependencyidentifies discardable pictures being coded pictures associated with anal_ref_idc field equal to 0 or equal to non-zero; determining whetherto decode or discard each of the coded pictures based on where the codedpicture is located within the one or more levels in the hierarchy ofdata dependency; and not decoding the coded pictures in the bitstreamthat are determined for discarding.
 2. The method of claim 1, thehierarchy of data dependency including at least one level ofdiscardability for reference pictures.
 3. The method of claim 1, whereinthe at least one bitstream header element is a two-bit syntax element.4. The method of claim 1, further comprising interpreting one or more ofthe values of the bitstream header element to determine a coded picturetype or slice types of the at least one coded picture.
 5. The method ofclaim 4, further comprising interpreting one or more of the values ofthe bitstream header element to determine whether one or more slices ofthe coded picture is a reference slice.
 6. The method of claim 1,further comprising interpreting one or more of the values of thebitstream header element to determine whether any part of the codedpicture is identified as a reference in a decoding process for decodingone or more other coded pictures in the bitstream.
 7. The method ofclaim 1, wherein for a bitstream header element nal_ref_idc having avalue that does not equal 0, the hierarchy of data dependency identifiesone or more levels indicating discardable picture slices.
 8. The methodof claim 1, wherein for a coded picture having a bitstream headerelement nal_ref_idc that does not equal 0, the hierarchy of datadependency identifies one or more levels indicating whether B slices inthe coded picture are discardable.